{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "# Deep Dream\n",
    "\n",
    "**Author:** [fchollet](https://twitter.com/fchollet)<br>\n",
    "**Date created:** 2016/01/13<br>\n",
    "**Last modified:** 2020/05/02<br>\n",
    "**Description:** Generating Deep Dreams with Keras."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Introduction\n",
    "\n",
    "\"Deep dream\" is an image-filtering technique which consists of taking an image\n",
    "classification model, and running gradient ascent over an input image to\n",
    "try to maximize the activations of specific layers (and sometimes, specific units in\n",
    "specific layers) for this input. It produces hallucination-like visuals.\n",
    "\n",
    "It was first introduced by Alexander Mordvintsev from Google in July 2015.\n",
    "\n",
    "Process:\n",
    "\n",
    "- Load the original image.\n",
    "- Define a number of processing scales (\"octaves\"),\n",
    "from smallest to largest.\n",
    "- Resize the original image to the smallest scale.\n",
    "- For every scale, starting with the smallest (i.e. current one):\n",
    "    - Run gradient ascent\n",
    "    - Upscale image to the next scale\n",
    "    - Reinject the detail that was lost at upscaling time\n",
    "- Stop when we are back to the original size.\n",
    "To obtain the detail lost during upscaling, we simply\n",
    "take the original image, shrink it down, upscale it,\n",
    "and compare the result to the (resized) original image.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Setup\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras.applications import inception_v3\n",
    "\n",
    "base_image_path = keras.utils.get_file(\"sky.jpg\", \"https://i.imgur.com/aGBdQyK.jpg\")\n",
    "result_prefix = \"sky_dream\"\n",
    "\n",
    "# These are the names of the layers\n",
    "# for which we try to maximize activation,\n",
    "# as well as their weight in the final loss\n",
    "# we try to maximize.\n",
    "# You can tweak these setting to obtain new visual effects.\n",
    "layer_settings = {\n",
    "    \"mixed4\": 1.0,\n",
    "    \"mixed5\": 1.5,\n",
    "    \"mixed6\": 2.0,\n",
    "    \"mixed7\": 2.5,\n",
    "}\n",
    "\n",
    "# Playing with these hyperparameters will also allow you to achieve new effects\n",
    "step = 0.01  # Gradient ascent step size\n",
    "num_octave = 3  # Number of scales at which to run gradient ascent\n",
    "octave_scale = 1.4  # Size ratio between scales\n",
    "iterations = 20  # Number of ascent steps per scale\n",
    "max_loss = 15.0\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "This is our base image:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "from IPython.display import Image, display\n",
    "\n",
    "display(Image(base_image_path))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "Let's set up some image preprocessing/deprocessing utilities:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "\n",
    "def preprocess_image(image_path):\n",
    "    # Util function to open, resize and format pictures\n",
    "    # into appropriate arrays.\n",
    "    img = keras.preprocessing.image.load_img(image_path)\n",
    "    img = keras.preprocessing.image.img_to_array(img)\n",
    "    img = np.expand_dims(img, axis=0)\n",
    "    img = inception_v3.preprocess_input(img)\n",
    "    return img\n",
    "\n",
    "\n",
    "def deprocess_image(x):\n",
    "    # Util function to convert a NumPy array into a valid image.\n",
    "    x = x.reshape((x.shape[1], x.shape[2], 3))\n",
    "    # Undo inception v3 preprocessing\n",
    "    x /= 2.0\n",
    "    x += 0.5\n",
    "    x *= 255.0\n",
    "    # Convert to uint8 and clip to the valid range [0, 255]\n",
    "    x = np.clip(x, 0, 255).astype(\"uint8\")\n",
    "    return x\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Compute the Deep Dream loss\n",
    "\n",
    "First, build a feature extraction model to retrieve the activations of our target layers\n",
    "given an input image.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "# Build an InceptionV3 model loaded with pre-trained ImageNet weights\n",
    "model = inception_v3.InceptionV3(weights=\"imagenet\", include_top=False)\n",
    "\n",
    "# Get the symbolic outputs of each \"key\" layer (we gave them unique names).\n",
    "outputs_dict = dict(\n",
    "    [\n",
    "        (layer.name, layer.output)\n",
    "        for layer in [model.get_layer(name) for name in layer_settings.keys()]\n",
    "    ]\n",
    ")\n",
    "\n",
    "# Set up a model that returns the activation values for every target layer\n",
    "# (as a dict)\n",
    "feature_extractor = keras.Model(inputs=model.inputs, outputs=outputs_dict)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "The actual loss computation is very simple:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "\n",
    "def compute_loss(input_image):\n",
    "    features = feature_extractor(input_image)\n",
    "    # Initialize the loss\n",
    "    loss = tf.zeros(shape=())\n",
    "    for name in features.keys():\n",
    "        coeff = layer_settings[name]\n",
    "        activation = features[name]\n",
    "        # We avoid border artifacts by only involving non-border pixels in the loss.\n",
    "        scaling = tf.reduce_prod(tf.cast(tf.shape(activation), \"float32\"))\n",
    "        loss += coeff * tf.reduce_sum(tf.square(activation[:, 2:-2, 2:-2, :])) / scaling\n",
    "    return loss\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Set up the gradient ascent loop for one octave\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "\n",
    "@tf.function\n",
    "def gradient_ascent_step(img, learning_rate):\n",
    "    with tf.GradientTape() as tape:\n",
    "        tape.watch(img)\n",
    "        loss = compute_loss(img)\n",
    "    # Compute gradients.\n",
    "    grads = tape.gradient(loss, img)\n",
    "    # Normalize gradients.\n",
    "    grads /= tf.maximum(tf.reduce_mean(tf.abs(grads)), 1e-6)\n",
    "    img += learning_rate * grads\n",
    "    return loss, img\n",
    "\n",
    "\n",
    "def gradient_ascent_loop(img, iterations, learning_rate, max_loss=None):\n",
    "    for i in range(iterations):\n",
    "        loss, img = gradient_ascent_step(img, learning_rate)\n",
    "        if max_loss is not None and loss > max_loss:\n",
    "            break\n",
    "        print(\"... Loss value at step %d: %.2f\" % (i, loss))\n",
    "    return img\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Run the training loop, iterating over different octaves\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "original_img = preprocess_image(base_image_path)\n",
    "original_shape = original_img.shape[1:3]\n",
    "\n",
    "successive_shapes = [original_shape]\n",
    "for i in range(1, num_octave):\n",
    "    shape = tuple([int(dim / (octave_scale ** i)) for dim in original_shape])\n",
    "    successive_shapes.append(shape)\n",
    "successive_shapes = successive_shapes[::-1]\n",
    "shrunk_original_img = tf.image.resize(original_img, successive_shapes[0])\n",
    "\n",
    "img = tf.identity(original_img)  # Make a copy\n",
    "for i, shape in enumerate(successive_shapes):\n",
    "    print(\"Processing octave %d with shape %s\" % (i, shape))\n",
    "    img = tf.image.resize(img, shape)\n",
    "    img = gradient_ascent_loop(\n",
    "        img, iterations=iterations, learning_rate=step, max_loss=max_loss\n",
    "    )\n",
    "    upscaled_shrunk_original_img = tf.image.resize(shrunk_original_img, shape)\n",
    "    same_size_original = tf.image.resize(original_img, shape)\n",
    "    lost_detail = same_size_original - upscaled_shrunk_original_img\n",
    "\n",
    "    img += lost_detail\n",
    "    shrunk_original_img = tf.image.resize(original_img, shape)\n",
    "\n",
    "keras.preprocessing.image.save_img(result_prefix + \".png\", deprocess_image(img.numpy()))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "Display the result.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "display(Image(result_prefix + \".png\"))\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "deep_dream",
   "private_outputs": false,
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}